Papers with sequence-to-sequence tasks
Enabling Real-time Neural IME with Incremental Vocabulary Selection (N19-2)
Copied to clipboard
| Challenge: | Input method editor (IME) converts sequential alphabet key inputs to words in a target language. |
| Approach: | They propose a neural-based language model that incrementally builds a subset vocabulary from the word lattice. |
| Outcome: | The proposed approach achieves 50x speedup on Japanese IME benchmark without losing conversion accuracy. |
Making a Point: Pointer-Generator Transformers for Disjoint Vocabularies (2020.aacl-srw)
Copied to clipboard
| Challenge: | Existing neural models rely on an overlap between source and target vocabularies to perform sequence-to-sequence tasks. |
| Approach: | They propose a pointer-generator transformer model for disjoint vocabularies that does not rely on an overlap between source and target vocs. |
| Outcome: | The proposed model outperforms a standard pointer-generator transformer by an average of 5.1 WER over 15 languages. |
Semantic Matching for Sequence-to-Sequence Learning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for sequence-to-sequence semantic matching do not match target sentences . Optimal partial transport (OPT) is a technique that partially matches semantically meaningful words between source and partial target sequences. |
| Approach: | They propose a semantic matching scheme based on the Optimal Partial Transport (OPT) to match semantically meaningful words between source and partial target sequences. |
| Outcome: | Extensive experiments show that the proposed approach improves over sequence-to-sequence models. |
LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models often require significant computational resources, often constraining input word or code token lengths. |
| Approach: | They propose to use the encoder-decoder attention scores to represent the importance of a code token across multiple contexts to reduce training and prediction time. |
| Outcome: | The proposed approach outperforms the SOTAs DietCode and SlimCode in code search and summarization tasks. |
Learning To Retrieve Prompts for In-Context Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | In-context learning is a new paradigm in natural language understanding . large pre-trained language models can be expensive to update . |
| Approach: | They propose an efficient method for retrieving training examples as prompts from annotated data and an LM. |
| Outcome: | The proposed method outperforms prior work and multiple baselines on three sequence-to-sequence tasks. |
Noise Isn’t Always Negative: Countering Exposure Bias in Sequence-to-Sequence Inflection Models (2020.coling-main)
Copied to clipboard
| Challenge: | Morphological inflection is a sequence-to-sequence task that sees great performance when data is plentiful, but performance falls off sharply in lower-data settings. |
| Approach: | They hypothesize that teacher forcing increases the likelihood that a model too closely models its training data. |
| Outcome: | Experiments show that teacher forcing can overfit models when they enter unknown territory. |
SyntaxShap: Syntax-aware Explainability Method for Text Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for explaining sequence-to-sequence tasks are not suitable for textual data. |
| Approach: | They propose a local, model-agnostic explainability method that takes into account the syntax in text data and extends Shapley values to account for parsing-based syntactic dependencies. |
| Outcome: | The proposed method is compared to state-of-the-art explainability methods for text generation tasks using various metrics including faithfulness, coherency, and semantic alignment of the explanations to the model. |
A Framework for Bidirectional Decoding: Case Study in Morphological Inflection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing encoder-decoders that generate sequences from left to right are prone to errors due to the "snowballing" effect. |
| Approach: | They propose a transformer-based encoder-decoder framework that produces sequences from the "outside-in" they argue that this approach is more principled than prior bidirectional decoders . |
| Outcome: | The proposed model beats the current system by over 4.7 and 2.7 points in accuracy on 2022 and 2023 tasks. |
The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | cipher decoding, riddles, code execution embedded into model prompts bypass safety safeguards of large language models (LLMs) . |
| Approach: | They introduce a novel class of adversarial jailbreak adversarials on large language models, termed Task-in-Prompt (TIP) attacks. |
| Outcome: | The proposed techniques circumvent safeguards in six state-of-the-art language models, including GPT-4o and LLaMA 3.2, and consistently generate restricted content . |
On Biasing Transformer Attention Towards Monotonicity (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work has focused on learning monotonic attention behavior via specialized attention functions or pretraining. |
| Approach: | They introduce a monotonicity loss function compatible with standard attention mechanisms and test it on sequence-to-sequence tasks. |
| Outcome: | The proposed monotonicity loss function can achieve largely monotonic behavior on grapheme-to-phoneme conversion, morphological inflection, transliteration, and dialect normalization tasks. |
E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | E-LANG is an efficient method for building large and highly capable language models . existing methods are only applicable to encoder-only backbones and classification tasks . |
| Approach: | They propose an efficient dynamic inference approach which distributes inference between large accurate Super-models and light-weight Swift models. |
| Outcome: | The proposed method outperforms existing methods on GLUE, SuperGLUE and WMT with 3.3X computation speed and 2.9X computation cost. |
Warmup Generations: A Task-Agnostic Approach for Guiding Sequence-to-Sequence Learning with Unsupervised Initial State Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing supervised fine-tuning (SFT) methods focus on directly generating the target output without leveraging the benefits of intermediate steps or initial guidance. |
| Approach: | They propose a task-agnostic framework that enables models to generate intermediate "warmup" sequences that are iteratively refined to maximize their contribution to the final output. |
| Outcome: | The proposed framework outperforms traditional supervised fine-tuning methods on translation, summarization, and multi-choice question answering tasks. |
Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Autoregressive models trained with maximum likelihood estimation suffer from exposure bias, i.e. the discrepancy between ground-truth prefixes used during training and model-generated prefix at inference time. |
| Approach: | They propose to use Elastic Weight Consolidation to better balance mitigating exposure bias with retaining performance. |
| Outcome: | The proposed method significantly outperforms maximum likelihood estimation and scheduled sampling baselines on four translation datasets. |
Sequence-to-Sequence Spanish Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Spanish language models have demonstrated proficiency in natural language understanding and generation, but there is a scarcity of encoder-decoder models specifically designed for sequence-to-sequence tasks. |
| Approach: | They propose to implement encoder-decoder architectures pre-trained on Spanish corpora . they use them to assess sequence-to-sequence tasks including summarization, question answering . |
| Outcome: | The proposed models outperform models on sequence-to-sequence tasks in Spanish . the models show that they perform well across all tasks, the authors note . |
Is linguistically-motivated data augmentation worth it? (2025.acl-long)
Copied to clipboard
| Challenge: | Data augmentation is a widely-used technique for addressing data scarcity. |
| Approach: | They compare linguistically-motivated and linguisticly-naive data augmentation strategies for two low-resource languages with different morphological properties. |
| Outcome: | The proposed methods produce synthetic data that follows all linguistic constraints, but they require linguistic expertise and are more difficult to implement. |